Sequential Labeling with online Deep Learning

In this paper, we leverage both deep learning and conditional random fields (CRFs) for sequential labeling. More specifically, we propose a mixture objective function to predict labels either independent or correlated in the sequential patterns. We learn model parameters in a simple but effective way. In particular, we pretrain the deep structure with greedy layer-wise restricted Boltzmann machines (RBMs), followed with an independent label learning step. Finally, we update the whole model with an online learning algorithm, a mixture of perceptron training and stochastic gradient descent to estimate parameters. We test our model on different challenge tasks, and show that this simple learning algorithm yields the state of the art results.